Advertisement insertion into media content for streaming

ABSTRACT

Methods and systems for advertisement insertion into streaming media content are described. In one method, an advertisement insertion tool receives a first indication of a start time of an advertisement break to be inserted into a stream of slices of media content. The advertisement insertion tool identifies a first slice of the slices that encompasses the start time of the advertisement break. The first slice comprises video frames. The advertisement insertion tool truncates the first slice to contain the video frames up to the start time of the advertisement break. An advertisement can be inserted after the truncated first slice and before a next second slice in the stream.

RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional No. 61/598,519,filed Feb. 14, 2012, the entire contents of which are incorporated byreference. This application is related to U.S. Ser. No. 13/764,536, andU.S. Ser. No. 13/764,563, filed concurrently with the presentapplication. The entire contents of which are incorporated by reference.

TECHNICAL FIELD

Embodiments of the present invention relate to the field of delivery ofmedia content over a network. Specifically, the embodiments of thepresent invention relate to advertisement insertion into streaming mediacontent.

BACKGROUND

The Internet is becoming a primary method for distributing media content(e.g., video and audio or just audio or just video) and otherinformation to end users. It is currently possible to download music,video, games and other media information to computers, cell phones, andvirtually any network capable device. The percentage of people accessingthe Internet for media content is growing rapidly. The quality of theviewer experience is a key barrier to the growth of video viewingon-line. Consumer expectations for online video are set by theirtelevision and movie viewing experiences.

Audience numbers for streaming video on the web are rapidly growing, andthere are a growing interest and demand for viewing video on theInternet. Streaming of data files or “streaming media” refers totechnology that delivers sequential media content at a rate sufficientto present the media to a user at the originally anticipated playbackspeed without significant interruption. Unlike downloaded data of amedia file, streamed data may be stored in memory until the data isplayed back and then subsequently deleted after a specified amount oftime has passed.

The process for preparing media content for adaptive streaming typicallyinvolves taking a source file, such as a movie, or a source feed, suchas a broadcast signal, and splitting it into temporal chunks or slicesof media (e.g., video and audio), and encoding each slice of the mediaat different quality levels. Typically the different quality levels areachieved by encoding the media content at different bit rates accordingto encoding profiles. During playback, and as network conditionsfluctuate, a consuming application selects a slice at one of thedifferent quality levels that is best suited for the present networkthroughput, and the consuming applications adjusts the quality up ordown as available bandwidth increases or decreases by selectingsubsequent slices at the appropriate quality levels.

Streaming media content over the Internet has some challenges, ascompared to regular broadcasts over the air, satellite or cable.Although the adaptive streaming approach provides many advantages, onemajor disadvantage is inserting advertisements. Conventionally, in thecase of live broadcast television, the broadcaster is in completecontrol of timing of the insertion and playback of advertisements. Forexample, if a 30-second advertisement is to be inserted, theadvertisement is selected and scheduled beforehand, and theadvertisement starts and ends immediately at the scheduled time. This ispossible because the broadcaster is in complete control of all theequipment and media content (regular programming plus advertisements)involved. In addition, all viewers see or hear the same advertisement atthe same time. The viewers are human beings who watch or listen to themedia content using a television or radio. When the time to insert anadvertisement arrives, the broadcaster inserts an advertisement into theoutput signal. The advertisement includes content which was previouslypositioned in the video equipment, which is capable of delivering thecontent into the output signal at the precise instant that it is calledfor, with very high reliability. In effect, all viewers see the sameadvertising at the same point in the media content, such as that of alive event or a scheduled programming.

As for Internet-based media content, one conventional approach insertsadvertising into the video stream at some point no later than the mediaserver or the equivalent, meaning the advertisements are encoded as anintegral part of the media content. Like conventional broadcasting,using this conventional approach, all viewers see the same advertisingat the same point in the media content, such as that of a live event,and a media player, operating on a networked device, is not involved inselecting the advertisements. The media player is a piece of softwarethat plays the media content (e.g., displays video and plays audio), andmay be a standalone software application, a web browser plug-in, acombination of browser plug-in and supporting web page logic, or thelike.

In the case of managing advertisements for on-demand, Internet-basedmedia content, sometimes referred to as video-on-demand (VOD), thetiming for advertisement insertion may be very flexible. For example,the VOD content may be a previously-recorded broadcast, as opposed tomedia content from a “live” broadcast. In this context, “live” meansthat the media content is anchored to a particular point in time for thepurpose of playing the media content. That is, the beginning of thepiece of content is scheduled to be played by a media player at aparticular point in time. In some cases, the media content of the liveevent is played out at the same time that the content is being encoded,and the anchor point is the time that the encoding of the event began.In other cases, the media content of a live event may be pre-encoded,but scheduled to play out at a particular point in time. In either case,the playback of the content is required to occupy approximately the sameamount of time as the original live event, in order to keep the playbackof the media content synchronized with the schedule. For live playback,the duration of advertisement breaks during playback should approximatethe duration of the advertisement breaks as they were encoded. Liveplayback (sometimes referred to as live playout) differs from VODplayback in that, with VOD content, there is no anchor point in realtime. A piece of VOD content begins playing when a viewer requests theVOD content. There is no requirement that the duration of advertisementbreaks be the same as the duration of the original advertisement breaks,during the playback of the VOD content. In fact, it is common for theduration of the advertisement breaks to be quite different in the VODcontent than from the original broadcast (e.g., for television). Forexample, an original television broadcast may be shown at a particularpoint in time, and subsequently, the publisher makes the televisionbroadcast available as VOD content, but without the originaladvertisements in the original television broadcast. Since the VODcontent is not anchored to a particular schedule, the advertisements inthe VOD content do not have to match the duration of the originaladvertisements.

For VOD advertisement insertion, advertisement breaks are typicallymarked at particular points in the video content, although theirlocations might also be computed based on the amount of time the viewerhas been viewing or by some other means. Advertisement breaks are notrequired to occupy any particular duration of time. In particular, theadvertisements in the VOD content do not have to match the originaladvertisement breaks in the program content. In fact, the number andduration of advertisements played during a particular advertisementbreak in the VOD content may vary from viewer to viewer.

For VOD advertisement insertion, when playback reaches an advertisementbreak, playback of the program content is suspended while theadvertisements are played. If there are brief pauses as advertisementsare fetched, or even during the playback of an advertisement, theviewer's experience during the advertisement break may be affected, butthis has no bearing on the viewer's experience after the program contentresumes. For example, in a VOD scenario, there is no concept of “lagbehind live,” because the event is pre-recorded and is available in itsentirety before playback begins. The term “lag behind live,” as usedherein, refers to the difference between when a particular piece ofcontent is encoded for Internet distribution and when a particularviewer views the piece of content on the media player. The “lag behindlive” can vary between viewers and can vary over time for a singleviewer. For a particular live event, delivered by a particulartechnology, there is a minimum lag behind live due to the time torecord, or otherwise capture the live event and to make the recordedcontent available for Internet distribution, for example.

The concept of “lag behind live” only applies to live events, not topre-recorded media content, unless the pre-recorded media content isscheduled to be played back at a particular point in time. In effect,for VOD content, there is no live point to which the viewer should berestored after an advertisement break. Advertisement breaks for VODcontent can be of any duration without affecting the viewer's experienceof the regular program content.

Since advertising is inserted into the live video stream of a live eventbefore the media server in one conventional system, all viewers seeingthe live event see the same advertisement at the same time. Thepublisher cannot control how many viewers see the advertisements. Ingeneral, there is no good way to take advantage of knowing how manyviewers see the advertisements. In particular, it is difficult orimpossible to be paid a higher price if a higher-than-expected number ofviewers see the advertisements. It is difficult or impossible to targetadvertising to particular demographics or geographic areas withoutsetting up a one or more media servers for each demographic orgeographic area and ensure that each viewer connects to the appropriatemedia server.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention is illustrated by way of example, and not by wayof limitation, in the figures of the accompanying drawings in which likereferences indicate similar elements. It should be noted that differentreferences to “an” or “one” embodiment in this disclosure are notnecessarily to the same embodiment, and such references mean at leastone.

FIG. 1 is a block diagram of exemplary network architecture in whichembodiments of an advertisement insertion tool may operate.

FIG. 2 is a block diagram of the advertisement insertion tool to insertadvertisements into media content for streaming according to oneembodiment.

FIG. 3A is a diagram that illustrates a flow of operation of theadvertisement insertion tool to create slices to accommodateadvertisement insertions according to one embodiment.

FIG. 3B is a diagram that illustrates a flow of operation of anadvertisement insertion tool to truncate slices to accommodateadvertisement insertions into the media content according to oneembodiment.

FIG. 4A is a flow diagram of one embodiment of a method of creating newslices to accommodate advertisement insertion.

FIG. 4B is a flow diagram of one embodiment of a method of preparing astream for inserting one or more slices of an advertisement whenslicing.

FIG. 4C is a flow diagram of one embodiment of a method of inserting theone or more slices of an advertisement into a stream for streaming.

FIG. 5A illustrates a stream of slices of media content before and afteradvertisement insertion according to one embodiment.

FIG. 5B illustrates a stream of slices of media content before and afteradvertisement insertion according to another embodiment.

FIG. 6 illustrates a diagrammatic representation of a machine in theexemplary form of a computing system for advertisement insertion.

DETAILED DESCRIPTION

A method and system for advertisement insertion into media content forstreaming is described. In the following description, numerous detailsare set forth. It will be apparent, however, to one of ordinary skill inthe art having the benefit of this disclosure, that embodiments of thepresent invention may be practiced without these specific details. Insome instances, well-known structures and devices are shown in blockdiagram form, rather than in detail, in order to avoid obscuring theembodiments of the present invention.

Some portions of the detailed description that follow are presented interms of algorithms and symbolic representations of operations on databits within a computer memory. These algorithmic descriptions andrepresentations are the means used by those skilled in the dataprocessing arts to most effectively convey the substance of their workto others skilled in the art. An algorithm is here, and generally,conceived to be a self-consistent sequence of steps leading to a desiredresult. The steps are those requiring physical manipulations of physicalquantities. Usually, though not necessarily, these quantities take theform of electrical or magnetic signals capable of being stored,transferred, combined, compared, and otherwise manipulated. It hasproven convenient at times, principally for reasons of common usage, torefer to these signals as bits, values, elements, symbols, characters,terms, numbers or the like.

It should be borne in mind, however, that all of these and similar termsare to be associated with the appropriate physical quantities and aremerely convenient labels applied to these quantities. Unlessspecifically stated otherwise as apparent from the following discussion,it is appreciated that throughout the description, discussions utilizingterms such as “receiving,” “generating,” “communicating,” “capturing,”“executing,” “defining,” “specifying,” “creating,” “recreating,”“processing,” “providing,” “computing,” “calculating,” “determining,”“displaying,” or the like, refer to the actions and processes of acomputing system, or similar electronic computing systems, thatmanipulates and transforms data represented as physical (e.g.,electronic) quantities within the computing system's registers andmemories into other data similarly represented as physical quantitieswithin the computing system memories or registers or other suchinformation storage, transmission or display devices.

Embodiments of the present invention also relate to an apparatus forperforming the operations herein. This apparatus may be speciallyconstructed for the required purposes, or it may comprise ageneral-purpose computing system specifically programmed by a computerprogram stored in the computing system. Such a computer program may bestored in a computer-readable storage medium, such as, but not limitedto, any type of disk including optical disks, CD-ROMs andmagnetic-optical disks, read-only memories (ROMs), random accessmemories (RAMs), EPROMs, EEPROMs, magnetic or optical cards, or any typeof media suitable for storing electronic instructions.

FIG. 1 is a block diagram of exemplary network architecture 100 in whichembodiments of an advertisement insertion tool 108 may operate. Thenetwork architecture 100 may include multiple client computing systems102, a computing system 104, and one or more server computing systems116, such as at a data center 130 coupled via a data communicationsnetwork 103 (e.g., public network such as the Internet or privatenetwork such as a local area network (LAN)). In the depicted embodiment,the media content 101 is captured at an event facility 120 and fed intothe computing system 104 that uses a slicer 106 to slice the mediacontent 101 into sliced media content 105. In one embodiment, the mediacontent 101 is a content file. The content file may include datacorresponding to video and/or audio corresponding to a televisionbroadcast, sporting event, movie, music, concert or the like. Theoriginal content file may include uncompressed video and audio, oralternatively, uncompressed video or audio. Alternatively, the contentfile may include compressed content (e.g., video and/or audio) usingstandard or proprietary encoding schemes. The original content file fromthe publisher may be digital in form and may include media contenthaving a high bit rate, such as, for example, 2 Mbps or greater. Inanother embodiment, the media content 101 may be a feed. The feed mayinclude data corresponding to video and/or audio.

In one embodiment, the slicer 106 may be a lightweight piece of softwarethat runs on the computing system 104 near the signal source (e.g.,source file or live feed), such as a laptop computer at the live eventor at a publisher's facility. The slicer 106 divides the media content101 into the temporal chunks (sliced media content 105) and pushes eachslice to a remote server computing system 116 in the data centerfacility 130. The server computing systems 116 may be one or moreencoders 124 that can process the individual slices. For example, theencoders 124 may retrieve a slice, encode the slice according to themultiple quality levels, and store them in the data storage 140 asencoded media content 142 to be delivered to the one or more clients 102via a web server 122, for example, using adaptive streaming techniques.

In this embodiment, the process of dividing a signal into temporalslices is decoupled from the work of encoding the slices to the variousbitrates and delivering them using the web server 122. Because theslicer 106 is not doing the work of encoding to the many output formats,it does not need large amounts of computing hardware to run, so thereare not large space and electricity requirements at the event facility120. There only needs to exist sufficient computing hardware to capturethe live signal (or source file) and divide it into the temporal slices.It is feasible that this slicing could even be performed on a mobiledevice with a camera, such as on the iPhone® mobile digital device.Alternatively, other types of portable or non-portable computing devicesmay be used to capture the signal source, or receive the signal sourcefrom another capture device, and slice the signal before sending to thedata center for encoding as would be appreciated by one of ordinaryskill in the art having the benefit of this disclosure.

In other embodiments, the encoders 124 could be employed at the eventfacility 120 and stored in the data storage 140 for delivery via the webserver 122 as described below. For example, in one embodiment, the livesignal is fed into encoding computers at the live event. These encodingcomputers divide the signal into temporal slices, encode the slices tothe various bitrates and upload them to a remote web server. One ofthese other embodiments may use a specialized delivery mechanism, like amicrowave or satellite link to deliver the encoded media content to thedata center 130. However, in the decoupled encoding embodiment describedabove, because the slicer 106 acquires the signal and creates thetemporal slices for delivery to the data center over the Internet, thespecialized delivery mechanism is not required. Instead, the slices canbe delivered over a normal Internet connection because there is only asingle slice per range of time rather than many versions of the slicesencoded at the various bitrates. Also, in the data center 130, nospecialized receiving hardware is required, because the slices from theslicer 106 are received over a standard internet connection. Encoders124 in the data center process the slices individually, create theencoded slices at the various bitrates, and store them to the datastorage 140 for delivery via the web server 122. Of course, the encodedmedia content 142 may be delivered to the clients 102 using more thanone web servers 122 at one or more locations as would be appreciated byone of ordinary skill in the art having the benefit of this disclosure.

In one embodiment, the web server 122 receives requests for encodedmedia content 142 over the network 103 from media players 113 executingon the client computing systems 102 (referred to herein as the“client”). The client 102 and the web server 122, which may be executedon a server of a content delivery network, may be coupled by the network103. The network 103 may include the Internet and network connections tothe Internet. Alternatively, the web server 122 and the client 102 maybe located on a common Local Area Network (LAN), Personal area network(PAN), Campus Area Network (CAN), Metropolitan area network (MAN), Widearea network (WAN), wireless local area network, cellular network,virtual local area network, or the like. The client 102 may be a clientworkstation, a server, a computer, a portable electronic device, anentertainment system configured to communicate over a network, such as aset-top box, a digital receiver, a digital television, a mobile phone orother electronic devices. For example, portable electronic devices mayinclude, but are not limited to, cellular phones, portable gamingsystems, portable computing devices or the like. The client 102 may haveaccess to the Internet via a firewall, a router or other packetswitching devices. The server computing systems 116 may be a networkappliance, a gateway, a personal computer, a desktop computer, aworkstation, etc.

In one embodiment, the computing environment 100 also includes one ormore content repositories, illustrated as a data storage device 140. Thedata storage device 140 may be a content repository in which encodedmedia content 142 may be stored. A publisher or a distributor of mediacontent may store the encoded media content 142 in the data storage 140.Media content from the publisher may be transferred over the network 103to the data storage 140 and served by the web server 122 (sometimesreferred to as the origin content server). In one embodiment, the webserver 122 delivers the media content to the client 102, which isconfigured to play the content on a media player 113 executing by aprocessing device of the client 102. The web server 122 delivers themedia content by streaming the media content to the client 102. In afurther embodiment, the client 102 is configured to receive differentportions of the media content from multiple locations.

In other embodiments, the encoded media content 142 from the web server122 may be replicated to other locations and served to the clients 102using other web servers 122 located on other server computing system.Alternatively, proxy cache servers can be used. For example, the client102 may request and receive content from any of the multiple web servers122, or proxy cache servers. In a further embodiment, the encoded mediacontent 142 may be forwarded directly to the other web servers 122 orproxy cache servers through direct communication channels without theneed to traverse the Internet. In another embodiment, the origin contentserver (web server 122), other web servers 122 and proxy cache serversmay be organized in a hierarchy of a CDN to deliver the media content tothe client 102. A CDN is a system of computers networked together acrossthe Internet that cooperates transparently to deliver content. In thedepicted embodiment, the web server 122 is executed on an origin contentserver. Alternatively, the web server 122 may be executed on one or moreother servers to deliver the encoded media content 142. The CDN may alsoinclude interconnected computer networks or nodes to deliver the mediacontent.

In the depicted embodiment, the data center 130 includes a contentmanagement system (CMS) 118 that manages the encoded media content 142,for example, using a database stored in data storage 140. In oneembodiment, the CMS 118 is a computing system including a memory and aprocessing device coupled to the memory, the processing device toexecute the advertisement insertion tool 108. In another embodiment, theadvertisement insertion tool 108 can be executed by a processing deviceof a computing system that is separate from the CMS 118. The CMS 118 mayalso manage the available encoded media content 142 using one or moredescriptor files 144. In the depicted embodiment, the publisher (orserver administrator) can create metadata descriptor files 144 using theCMS 118. The metadata descriptor files 144 can be fetched or pushed tothe media player(s) 113. In one embodiment, the metadata descriptor file144 is M3U file. A M3U file is a computer file format that stores mediaplaylists. Typically, a M3U file is a plain text file that specifies thelocations of one or more media files. Each entry of the M3U file carriesone specification, which can be an absolute local pathname, a localpathname relative to the M3U file location, or a URL (both absolute andrelative URLs). The M3U file can also include comments, which areprefaced by the ‘#’ character. In extended M3U, the “#” character mayalso introduce extended directives. The Unicode version of M3U is M3U8,which uses UTF-8 Unicode characters.

In one embodiment, the metadata descriptor file 144 gives an overview ofthe content and is initially requested by the media player 113 in orderto know how to request specific files or data ranges within a file. Forexample, a user may visit a webpage with a link to the video to bestreamed. When the link is activated, the media player fetches themetadata descriptor file 144 to determine what quality levels areavailable and the locations of the corresponding files or data rangeswithin a single file for adaptive streaming. It should be noted that theweb server sends transport stream files in response to the requests,however, the media content may be stored in one or multiple files.Regardless of how the media content is stored, the web server 122 sendsthe transport stream files with the corresponding slice of media contentrequested. When the media content is stored as discrete files, the webserver may send the discrete file (encoded according to the appropriatequality profile) as the transport stream file in response to therequest. However, when the media content is stored a single file, anidentifier and an offset may be used to retrieve the appropriate sliceof the file (encoded according to the appropriate quality profile), andthe web server 122 can send this slice as a transport stream file inresponse to the request.

In one embodiment, the M3U8 file is created with the available qualityprofiles and where the media content at those profiles are located eventhough the media content at those different profiles. The media player113 reads the descriptor file 144 for the media content that is beingrequested. The metadata descriptor file 144 may also include a uniquecontent identifier (UCID), duration of the media content, availablequality profiles and locations of where the media content can beretrieved. The quality profiles may specify parameters, such as widthand height of the image (i.e., image size), video bit rate (i.e., rateat which the video is encoded), audio bit rate, audio sample rate (i.e.,rate at which the audio is sampled when captured), number of audiotracks (e.g., mono, stereo, or the like), frame rate (e.g., frame persecond), staging size, or the like. For example, the media players 113may individually request different quality levels of the same mediacontent; for example, different media players 113 may request the sameportion (e.g., same time index) of the media content but at differentquality levels. For example, one media player may request a slice havingHD quality video, since the computing device of the requesting mediaplayer has sufficient computational power and sufficient networkbandwidth, while another media player may request a slice having a lowerquality, since its computing device may not have sufficient networkbandwidth, for example. In one embodiment, the media player 113 shiftsbetween quality levels at the slice boundaries by requesting slices fromdifferent copies (e.g., different quality streams) of the media content.Alternatively, the media player 113 can request the slices using othertechniques that would be appreciated by those of ordinary skill in theart having the benefit of this disclosure. The metadata descriptor file144 may also include other metadata, including, for example, air date ofthe content, title, actresses, actors, a start index, an end index,proprietary publisher data, encryption level, content duration, episodeor program name, publisher; available tools for the end-usernavigational environment, such as available menus, thumbnails, sidebars,advertising, fast-forward, rewind, pause, and play, or the like; orbit-rate values, including frame size, audio channel information,codecs, sample rate, and frame parser information. In anotherembodiment, other types of publishing systems other than CMS 118 may beused to create the metadata descriptor files 144. Also, the CMS 118 maybe hosted on another computing system within the data center 130 oroutside of the data center 130. For example, the data center 130 may bea cloud computing service provided by a third-party service provider andthe CMS 118 is executed on a computing device that can access the cloudcomputing service over a public or private network.

The CMS 118 may also allow a publisher to indicator where advertisementscan be inserted into the encoded media content 142 for streaming. In oneembodiment, the CMS 118 receives advertisement insertion input 109, suchas indications of the presentation times of when advertisements shouldstart and end or when the advertisements start and their duration. Theadvertisement insertion tool 108 may be software that executes on theCMS 118 and uses the advertisement insertion input 109 to insertadvertisements into the streaming encoded media content 142. Detailsregarding the operation of the advertisement insertion tool 108 aredescribed below with respect to FIGS. 2-3B.

In one embodiment, the CMS 118 can use the advertisement insertion tool108 to create new slices to accommodate an advertisement inserted atpresentation time that does not necessarily align with the sliceboundaries. The advertisement insertion process can be done on theslices before encoding by the encoder 124 or after encoding as would beappreciated by one of ordinary skill in the art having the benefit ofthis disclosure. The slices may have an equal duration, oralternatively, the slices may have equal file sizes, or vary one toanother according to other requirements. During operation, as describein more detail below, each portion of the encoded media content 142 maybe retrieved, decoded and re-encoded into an encoded representation ofthat portion of the media content at the specified quality level.Typically, in adaptive streaming, a media content file can be encodedinto multiple encoded representations according to different qualityprofiles and stored as separate files that are independently requestablefrom the data storage 140 via the web server 122. However, in theembodiments described herein, the media content can be stored in thedata storage 140 and encoded at the requested quality level, and theadvertisement insertion tool 108 can be used to insert advertisementsinto the streaming media content that is being played back on the mediaplayer 113. It should be noted that the advertisements can be added andencoded into the encoded media content 142 so that the advertisement ispart of the stream itself. In other embodiments, the advertisements areinserted into the descriptor files and the slices are updated toaccommodate advertisements at the designated locations. It should benoted that the duration of the set of one or more advertisementsinserted into a stream can match the duration of a set of originaladvertisements or can be shorter or longer than the duration of the setof original advertisements. Also, when the stream does not have anyoriginal advertisements, the set of one or more advertisements beinginserted can have any duration. Also, the actual advertisements and theduration of these advertisements can vary for the requesting users. Forexample, one advertisement may be streamed to a first media player whenreaching an advertisement marker in the streaming content and anotheradvertisement may be streamed to a second media player reaching the sameadvertisement marker in the streaming content. The decision(s) of whichadvertisement is streamed may be based on various factors, includinggeography, user profile, or the like.

In other embodiments, the computing environment 100 may include otherdevices, such as directory servers, Digital Rights Management (DRM)servers, statistic servers, devices of a network infrastructure operator(e.g., an ISP), or the like.

In one embodiment, the same server computing system 116 may execute theweb server 122 and the advertisement insertion tool 108, which isdescribed in more detail below. Alternatively, the functionality of theadvertisement insertion tool 108 and the web server 122 can bedistributed over the two machines. For example, in one embodiment, theserver computing system 116 may host just the advertisement insertiontool 108 and not the web server 122, and another server computing system(not illustrated) can host the web server 122 to handle requests for themedia content. Alternatively, other configurations are possible as wouldbe appreciated by one of ordinary skill in the art having the benefit ofthis disclosure.

This descriptor file 144 permits the media player 113 to request theslices, and the advertisement insertion tool 108 handles creating thenew or modified slices to accommodate the inserted advertisements fordelivery to the media player 113. From the media player's perspective,the media player 113 requests the slices according to the descriptorfile 144, which has been updated to include the advertisements orindications on when advertisements are to be played back duringstreaming of the slices.

FIG. 2 is a block diagram of the advertisement insertion tool 108 toinsert advertisements into media content for streaming according to oneembodiment. The advertisement insertion tool 108 may include anAdvertisement Decision Service (ADS) interface 221 that communicateswith an ADS 210, a container reader 222, an editor interface 223, aslice creator 224, an audio decoder 228, a video decoder 226, a videoencoder 232, an audio encoder 234, a video frame buffer 242, an audioframe buffer 244, and a descriptor file manager 246.

In the depicted embodiment, the ADS 210 is an external service runningon a remote computing system accessible over the Internet. The ADS 210selects the advertisement to play from an inventory of availableadvertisements (e.g., advertisement impressions). The inventory mayrepresent the advertisements which the publisher of the media contenthas sold to advertising customers. Rules in the ADS 210 may influencethe selection of an advertisement from the inventory. The rules may takeinto account information supplied in a request by a media player 113,such as the local time of player, keywords, the geographic location ofthe media player (e.g., determined by the host's IP address),demographics of the viewer (user of the media player), and the like.These rules and others would be appreciated by one of ordinary skill inthe art, and as such, additional description of these rules has not beenincluded. In one embodiment, when the media player 113 requests theadvertisement decision, as indicated in the descriptor file, theadvertisement insertion tool 108 may receive the request from the mediaplayer 113, request the advertisement from the ADS 210 and return theidentity of the advertisement to be displayed to the requesting mediaplayer 113. Alternatively, the media player 113 can request theadvertisement from the ADS 210 without using the advertisement insertiontool 108. Each individual media player 113 may potentially receive adifferent advertisement response for the same advertisement opportunity(e.g., the scheduled advertisement break). After the advertisement isselected in this fashion, the media player 113 fetches and plays theadvertisement.

In one embodiment, the content server stores the availableadvertisements, and the media player 113 requests the selectedadvertisements before the advertisement break. In another embodiment,the media player 113 stores the available advertisements. Alternatively,the available advertisements are stored and retrieved over the Internetfrom servers that are distinct from the media player computer and thecomputing system hosting the ADS 210. In one embodiment, the ADS 210supports an interface that receives the duration of the advertisementbreak from the media player 113 and/or any targeted-advertisementinformation, and returns the identities of advertisement(s) that shouldbe requested by the media player 113 to fill the advertisement break. Inother cases, the media player 113 determines the number and durations ofthe one or more advertisements selected to fill the scheduledadvertisement break, and makes individual requests for the one or moreadvertisements of the appropriate durations from the ADS 210, orelsewhere. Alternatively, other techniques for advertisement selectioncould be implemented, for example, the ADS 210 may be implemented withinthe media player 113, or within the content server. Alternatively, theselection of advertisements may be done using other techniques thanusing the ADS 210. The details of how advertisements could otherwise beselected to fill an advertisement break would be appreciated by one ofordinary skill in the art, and accordingly, a detailed description ofsuch has not been provided. It should be noted that the embodiments ofFIG. 2 describe advertisement insertion into streaming video.Alternatively, advertisement insertion may be performed by the mediaplayer 113 that is configured to receive the live media content usingother delivery methods than streaming.

In the depicted embodiment, the editor interface 223 communicates with aremote computing system 202 operated by a publisher or editor of theencoded media content. For example, an editor of the content can send aplay request 201 to the advertisement insertion tool 108 via the editorinterface 223 in order to playback the media content to select points toinsert advertisements. In response to the play request 201, theadvertisement insertion tool 108 sends the appropriate media content 203to the remote computing system 202. The editor may use a software toolon the remote computing system 202 to select the points in the mediacontent where advertisements should be inserted. The remote computingsystem 202 sends an advertisement insertion input 209 to theadvertisement insertion tool 108 via the editor interface 223. Theadvertisement insertion input 209 may include a presentation time ofwhen the advertisement should be inserted. For example, theadvertisement insertion input 209 may specify a presentation time for astart of an advertisement and a presentation time for an end of theadvertisement. Alternatively, the advertisement insertion input 209 mayspecify a presentation time and a duration of an advertisement. Theadvertisement insertion tool 108 uses the input 209 to create new slicesin the encoded media content 142 to accommodate advertisement insertionof an advertisement. The advertisement insertion tool 108, using theslice creator 224, creates the new slices in the encoded media content142, and updates the descriptor file(s) 144) using the descriptor filemanager 246 as described herein.

In the depicted embodiment, the container reader 222 can be used to readcontainers of the media content 142. For example, the media content filemay be an MP4 file, and the MP4 file is the container that storesmetadata and the actual audio and video samples. The actual audio andvideo samples have been encoded using a codec, such as AAC, H264 or thelike. Using the container reader 222, the advertisement insertion tool108 can read the metadata information from the container. Readingmetadata information from the container is not a computationallyintensive operation as compared to decoding the audio and video samples.

In one embodiment, using the container reader 222, the advertisementinsertion tool 108 can scan the media content file, looking for all thevideo key frames, for example. The advertisement insertion tool 108 mayread the encoded media content 142, using the container reader 222without actually decoding the encoded video frames and the encoded audiosamples. In effect, the container reader 222 can read a container(sometimes called a wrapper) to obtain metadata information, and canread the pass the video and audio samples to the decoder, but instructthe decoder not to decode, for example, to determine which video framesare key frames without using the computational resources to decode theactual video frames. Alternatively, the encoded video frames and audiosamples can be discarded before being sent to the decoders when locatingparticular key frames.

The container reader 222 may read the container, such as the AVIcontainer or MP4 container (or other types of containers, such as MP3,MKV, or the like), and may output the encoded video frames and theencoded audio samples, as well as any other metadata information in thecontainer. For example, an MP4 container may indicate that there are 2media tracks in this file, one being video the other audio, that thevideo is H264 encoded, and the audio is AAC in stereo. When thecontainer reader 222 outputs the frames and samples, the advertisementinsertion tool 108 can determine the track of the frame or sample,whether it is audio or video. The video frames being output may alsoinclude a flag that indicates that it is a key frame. The advertisementinsertion tool 108 can identify the type and duration of the samples andframes, including the presentation times and durations of the key framesof the media content. For example, the advertisement insertion tool 108can determine that a video frame has a presentation time of 20.2 secondsfrom the beginning and it is a key frame and that the correspondingaudio has 1024 audio samples. Although the actual data has not beenaccessed yet, the output of the container reader 222 may be split intotwo different pipelines, one pipeline for the audio codec (audio decoder228), and the other for the video codec (video decoder 226). Thedecoders, when instructed to decode, output the raw video frame and theraw audio samples into the video frame buffer 242 and the audio framebuffer 244, respectively. Thus, the advertisement insertion tool 108 caninstruct the decoders to not decode the video frames and the audiosamples when performing the pre-processing of the media content.However, when needed to create a new slice, for example, theadvertisement insertion tool 108 can instruct the decoders to decode thevideo frames and the audio samples so that the video frames and audiosamples can be re-encoded in the new slices. In another embodiment, theadvertisement insertion tool 108 may be configured to not send the videoframes and audio samples to the decoders when performing thepre-processing of the media content. In another embodiment, theadvertisement insertion tool 108 may be configured to instruct thedecoders to discard the video frames and audio samples. Alternatively,the advertisement insertion tool 108 can use a series of commands toturn the decoders on and off. When the decoders are off the video framesand audio frames may be passed through without being decoded and thensubsequently discarded. Alternatively, the container of the slice mayalready contain an index of where the video key frames are locatedwithin the slice. For example, this index can be created when adding themedia content file to the content library or at a time before receivingrequests from clients. Alternatively, this index can be created at thetime of receiving the request “on the fly,” if not previously processed.The advertisement insertion tool 108 can use the index in order tocreate one or more new slices in the encoded media content 142 toaccommodate advertisement insertion of an advertisement, as well as toupdate the descriptor file(s) 144.

The key frames have a corresponding time stamp and the advertisementinsertion tool 108 can use the time stamps of the key frames todetermine the relation between the key frames. For example, if there isa first key frame at 15 seconds into the media content and a second keyframe is at 33 seconds, and a 15-second advertisement is to be insertedat 20 seconds, the advertisement insertion tool 108 can create a newslice that includes the media content up to the 20 seconds, and cancreate a second new slice to transition back to the media content afterthe advertisement, as described below with respect to FIGS. 4A-4C. Forexample, if advertisement is to be inserted into a slice that containsthe 20 second mark, the new slice could contain the media content up tothe 20 second mark, and the frame after the 20 second mark could beconverted into a new key frame to create a new slice for playback afterthe advertisement insertion. In this embodiment, the advertisementinsertion tool 108 can start reading back at the key frame at the 15seconds in order to decode the video and audio samples up to the 20seconds to convert the frame at 20 seconds into a key frame for the newslice. The advertisement insertion tool 108, using the video decoder 226and audio decoder 228, decodes the encoded video frames and the encodedaudio samples to generate the raw video and the raw audio, respectively.These samples are stored in the video frame buffer 242 and audio framebuffer 244. The advertisement insertion tool 108 can discard the samplesthat are outside the time ranges of the new slices. The advertisementinsertion tool 108, using the video encoder 232, encodes the raw video(for the pertinent time range) into encoded video frames, and, using theaudio encoder 234, encodes the raw audio (for the pertinent time range)into encoded audio frames. The advertisement insertion tool 108 caninstruct the video encoder 232 and the audio encoder 234 of the qualityprofile to be used for encoding the audio and video. The advertisementinsertion tool 108, using the slice creator 224, creates the new sliceshaving the encoded video frames and the encoded audio frames for therespective time ranges, and stores the encoded media content 142 intothe data storage 140. After creation of the new slices, theadvertisement insertion tool 108 can update the descriptor file 144using the descriptor file manager 246 so that the media player requeststhe new slices and the advertisement. The updated descriptor files aredescribed and illustrated in tables below.

It should be noted that the advertisement insertion tool 108 may be usedin an adaptive streaming context, as well as other streaming contextsthat are not adaptive streaming.

The video and audio stored in the buffers 242 and 244 can then be usedto create a slice for the requested time range. In particular, the slicecreator 224 can instruct the audio encoder 234 and video encoder 232 toencode the raw data into encoded audio frames and video frames for theslice. The advertisement insertion tool 108 can instruct the slicecreator 224 to create the slice with the necessary encoded audio framesand video frames.

In another embodiment, when the first frame after the insertedadvertisement is not a key frame, the advertisement insertion tool 108can decode the frame as described above (decoding from the previous keyframe and any intervening delta frames), and then re-encode this firstframe as a key frame. The subsequent frames to this first frame may beable to be copied from the original data so that the advertisementinsertion tool 108 does not have to re-encode the subsequent deltaframes after the newly created key frame. This may speed up theoperation, and may not lose any quality by re-encoding the video frames.Alternatively, the advertisement insertion tool 108 can re-encode allvideo frames again as would be appreciated by one of ordinary skill inthe art having the benefit of this disclosure.

FIG. 3A is a diagram that illustrates a flow of operation of theadvertisement insertion tool to create slices to accommodateadvertisement insertions according to one embodiment. Advertisementinsertion, as used herein, refers to the process of preparing thestream(s) of the media content to insert one or more advertisements intothe stream(s) for streaming. A single stream can be prepared forinserting one or more advertisements at one or more designatedlocations. Alternatively, multiple streams, such as different qualitystreams of the same media content, can be prepared for inserting one ormore advertisements at one or more designated locations.

In one embodiment, when the advertisement insertion tool 108 receives anadvertisement insertion request 301 for an encoded media content 142(original content file), the advertisement insertion tool 108 canprocess the encoded media content 142. For example, the media contentfile may be an MP4 file. The MP4 file is the container that storesmetadata and the actual audio and video samples. The actual audio andvideo samples have been encoded using a codec, such as AAC, H264 or thelike. Using the container reader 222, the advertisement insertion tool108 can read the metadata information from the container. Readingmetadata information from the container is not a computationallyintensive operation as compared to decoding the audio and video samples.Using the container reader 222, the advertisement insertion tool 108 canscan the media content file, looking for all the video key frames. Inone embodiment, the advertisement insertion tool 108 can inspect one ormore original slices to locate key frames and to create new slices toaccommodate insertion of an advertisement at the desired presentationtimes as described in more detail below. In another embodiment, theadvertisement insertion tool 108 creates an index of the media contentfile, and the video key frames can become index points. This index canbe created when adding the media content file to the content library orat a time before receiving advertisement insertion requests 301 from apublisher. Alternatively, this index can be created at the time ofreceiving the advertisement insertion request 301 “on the fly,” if notpreviously processed.

The key frames have a corresponding time stamp and the advertisementinsertion tool 108 can use the time stamps of the key frames todetermine the relation between the key frames. For example, if there isa key frame at 15 seconds into the media content and the second keyframe is at 33 seconds, and an advertisement insertion request 301 comesin for inserting a 30-second advertisement at 25 seconds presentationtime (referred to as the advertisement insertion point), theadvertisement insertion tool 108 can start reading back at the key frameat the 15 seconds in order to decode the video and audio samples up tothe 25 seconds so that the advertisement insertion tool 108 can create anew slice containing the media content between 15 seconds and 25seconds. The advertisement insertion tool 108, using the video decoder226 and audio decoder 228, decodes the encoded video frames 333 and theencoded audio samples 335 to obtain the raw video 337 and the raw audio339, respectively. These samples are stored in the video frame buffer242 and audio frame buffer 244. The advertisement insertion tool 108 candiscard the samples that are after the presentation time for the newslice. In another embodiment, the advertisement insertion tool 108 doesnot decode the key frames and any intervening delta frames up to theadvertisement insertion point; rather, the advertisement insertion tool108 uses the data from the original slice and discards the remainingframes after the advertisement insertion point or marks these frames tobe discarded when decoded. The frames after the advertisement insertionpoint can be used for creation of a second new slice (when no blackframes are used in the media content to mark the advertisementinsertion). The subsequent frame after the advertisement insertion pointis converted into a key frame for the second new slice as described inmore detail below.

In some embodiments, to create the first new slice, the advertisementinsertion tool 108, using the video encoder 232, encodes the raw video337 for the designated time range (e.g., 15-25 seconds) into encodedvideo frames 341, and, using the audio encoder 234, encodes the rawaudio 339 for the designated time range (e.g., 15-25) into encoded audioframes 343. The advertisement insertion tool 108 can instruct the videoencoder 232 and the audio encoder 234 of the quality profile to be usedfor encoding the audio and video. The advertisement insertion tool 108,using the slice creator 224, creates a slice 345 having the encodedvideo frames 341 and the encoded audio frames 343 for the designatedtime range, and stores the slice 345. In a further embodiment, aresponse 303 is sent to the requesting computer to indicate the newslices created for the advertisement insertion request 301. Thisinformation can be used to update the descriptor file 144 as describedin more detail below.

In one embodiment, a new key frame may be created at the identified timeindex, even if the sample at that time was not a key frame previously.Continuing the example from above, the advertisement insertion tool 108,using the slice creator 224 may create a second new slice at 25 secondswith a key frame at 25 seconds. The second new slice can be played backafter the advertisement so that presentation time may be changed from 25seconds to 55 seconds, for example, when inserting a 30-secondadvertisement. The new slice 345 has a key frame in order for the mediaplayer 113 to be able to decode and play the media content after theadvertisement insertion.

In another embodiment, the advertisement insertion tool 108, which canaccess the media content with random access, finds the video frame atthe advertisement insertion point and creates two new slices; one slicebefore the advertisement insertion point and one slice after theinsertion point. In another embodiment, the media content alreadycontains black frames to designate where the advertisement is to beinserted in these cases, the first new slice is a portion of a firstoriginal slice before a start time of the black frames and the secondnew slice is a portion of a second original slice after an end time ofthe black frames. The creation of the second new slice allows the mediacontent to resume after the insertion point with the first frame of thenew slice being independently decodable. In another embodiment, themedia content does not contain black frames, but rather includesoriginal advertisements in an original feed. For example, anover-the-air TV feed may already have advertisements, but theseadvertisements were sold for over-the-air TV only. The advertisementinsertion tool 108 can receive a signal indicating when theadvertisement break starts and another signal indicating when theadvertisement break ends. The advertisement tool 108 can truncate thecurrent slice, discarding the video frames of the originaladvertisements until the signal indicating the end of the advertisementbreak is received, as described herein. At this point, the advertisementtool 108 starts with the next slice. In another embodiment, theadvertisement insertion tool 108, using the index can go back to themost recent key frame previous in time than the advertisement insertionpoint in order to create the first new slice. The advertisementinsertion tool 108 can also use the index to create the second newslice.

It should be noted that the advertisement insertion tool 108 may be usedin an adaptive streaming context, as well as other streaming contextsthat are not adaptive streaming. For example, if a user may be playing avideo on a mobile phone while in transit. The media player on the mobilephone will retrieve the metadata descriptor file and determine whatquality profiles are available for that video and where to retrieve thevideo. As described above, the metadata descriptor file may refer tofiles or portions of files that do not exist yet, and the advertisementinsertion tool 108 can receive requests from the media player fordifferent portions of the video at one or more different qualityprofiles. For example, the media player may start out requesting mediacontent that has been encoded at a 200 kb/s rate and determines that itcan get better, and start requesting portions of the media content at ahigher rate. As far as the mobile phone and media player is concerned,the media content has been encoded according to the different qualityprofiles for the adaptive streaming, and does not know that theadvertisement insertion tool 108 has created new slices to accommodatethe advertisement insertion.

In one embodiment, when doing the pre-processing of the media content,the advertisement insertion tool 108 can read the encoded media content142, using the container reader 222 without actually decoding theencoded video frames 333 and the encoded audio samples 335. In effect,the container reader 222 can read a container (sometimes called awrapper) to obtain metadata information, and can read the pass the videoand audio samples to the decoder, but instruct the decoder not todecode, for example, to determine which video frames are key frameswithout using the computational resources to decode the actual videoframes. Alternatively, the encoded video frames 333 and audio samples335 can be discarded before being sent to the decoders. In oneembodiment, there is a container and the container has the audio samplesand the video frames that are encoded. The container can be opened andread without using computational resources to decode the audio samplesand video frames, especially, since this pre-processing does not requiredecoded the video frames to obtain the actual pixel data or the audiosamples. These frames and samples can be discarded before the decodersor the decoders can be instructed not to decode in order to quicklyprocess the encoded media content 142. In one embodiment, thepre-processing may be done by a pipeline processing architecture asillustrated in FIG. 3A in which the advertisement insertion tool 108orchestrates the pre-processing. In one exemplary embodiment, theadvertisement insertion tool 108 fetches a file (or portion of a file)that has the encoded media content 142. The container reader 222 readsthe container, such as the AVI container or MP4 container (or othertypes of containers, such as MP3, MKV or the like). The container reader222 outputs the encoded video frames 333 and the encoded audio samples335, as well as any other metadata information in the container. Forexample, an MP4 container may indicate that there are 2 media tracks inthis file, one being video the other audio, that the video is H264encoded, and the audio is AAC in stereo. When the container reader 222outputs the frames and samples, the advertisement insertion tool 108 candetermine the track of the frame or sample, whether it is audio orvideo. The video frames being output may also include a flag thatindicates that it is a key frame. The advertisement insertion tool 108can identify the type and duration of the samples and frames. Forexample, the advertisement insertion tool 108 can determine that a videoframe has a presentation time of 20.2 seconds from the beginning and itis a key frame and that the corresponding audio has 1024 audio samples.Although the actual data has not been accessed yet, the output of thecontainer reader 222 splits into two different pipelines, one pipelinefor the audio codec (audio decoder 228), and the other for the videocodec (video decoder 226). The decoders, when instructed to decode,output the raw video frame 337 and the raw audio samples 339 into thevideo frame buffer 242 and the audio frame buffer 244, respectively.Thus, the advertisement insertion tool 108 can instruct the decoders tonot encode the video frames 333 and the audio samples 335 whenperforming the pre-processing. However, when servicing an advertisementinsertion request 301, the advertisement insertion tool 108 can instructthe decoders to decode the video frames 333 and the audio samples 335when needed to re-encode frames and samples for the new slices.

In another embodiment, the advertisement insertion tool 108 may beconfigured to not send the video frames and audio samples to thedecoders when performing the pre-processing. In another embodiment, theadvertisement insertion tool 108 may be configured to instruct thedecoders to discard the video frames and audio samples. Alternatively,the advertisement insertion tool 108 can use a series of commands toturn the decoders on and off. When the decoders are off the video framesand audio frames may be passed through without being decoded and thensubsequently discarded.

In one embodiment, the advertisement insertion tool 108 may record apresentation for all key frames and the location of the key frame withinthe actual file. This is done to build the index of these points thatcan be jumped to in order to start reading the data when decoding andencoding the data into the slice 345.

In another embodiment, when the preprocessing is done in connection withthe request, the advertisement insertion tool 108 can read the encodedmedia content 142 to determine the index points for the key frames. Whenthe advertisement insertion tool 108 needs to convert or re-encode anyparticular frame, the advertisement insertion tool 108 can decode thekey frame any subsequent delta frames for the designated time range, anddischarge the other frames. This way, the advertisement insertion tool108 does not use a lot of computational resources decoding all frames,but only those that are needed to create the new slices. These decodedframes can be stored in the video frame buffer 242. In anotherembodiment, the advertisement insertion tool 108 can discard everythinguntil it has identified the specific key frame needed to decode thefirst frame of the requested time range, and then go back and startdecoding from the key frame up through the requested time range. Thevideo and audio stored in the buffers 242 and 244 can then be used tocreate a slice for the requested time range. In particular, theadvertisement insertion tool 108 can instruct the audio encoder 234 andvideo encoder 232 to encode the raw data into encoded audio frames andvideo frames for the slice 345. The advertisement insertion tool 108 caninstruct the slice creator 224 to create the slice with the necessaryencoded audio frames and video frames.

In another embodiment, when the first frame of the new slice, such asthe second new slice after the advertisement, is not a key frame, theadvertisement insertion tool 108 can decode the frame as described above(decoding from the previous key frame and any intervening delta frames),and then re-encode this first frame as a key frame. The subsequentframes to this first frame may be able to be copied from the originaldata so that the advertisement insertion tool 108 does not have tore-encode the subsequent delta frames after the newly created key frame.This may speed up the operation, and may not lose any quality byre-encoding the video frames. Alternatively, the advertisement insertiontool 108 can re-encode all video frames again as would be appreciated byone of ordinary skill in the art having the benefit of this disclosure.In some embodiments, the advertisement insertion tool 108 may create anindex so that frames that are not independently decodable (delta frames)can be decoded and encoded as a key frame that is independentlydecodable. The subsequent delta frames may be encoded again or may becopied from the original data.

FIG. 3B is a diagram that illustrates a flow of operation of anadvertisement insertion tool 308 to truncate slices to accommodateadvertisement insertions into the media content according to oneembodiment. The advertisement insertion tool 308 works in connectionwith the slicer 106. The advertisement insertion tool 106 may reside onthe same computing system 104 as the slicer 306 at the event facility.Alternatively, the advertisement tool 106 may reside on a differentcomputing system than the computing system that executes the slicer 106.As described above, the slicer 106 receives the media content 101 andgenerates slices of the media content, outputting the sliced mediacontent 105. The sliced media content 105 can be subsequently encoded asdescribed herein. The details of slicing by the slicer 106 are notrepeated here. In this embodiment, the advertisement tool 308 can workin connection with the slicer 106 to slice the media content and totruncate one or more slices to accommodate an advertisement break.

In one embodiment, the advertisement insertion tool 308 receives anindication of an advertisement break 351. This indication may be a starttime indicating where in the media content one or more advertisementsneed to be inserted into the stream. In one embodiment, the indicationalso indicates a duration of the advertisement break 351. Alternatively,the advertisement insertion tool 308 may receive a second indication ofthe end of the advertisement break. Alternatively, the indication of theadvertisement break 351 may indicate a start time and an end time of theadvertisement break 351. The advertisement insertion tool 308 identifiesa first slice that encompasses the start time of the advertisement break351. For example, while slicing the media content 106, the advertisementinsertion tool 308 receives a signal that an advertisement is to beinserted at a specified presentation time of the media content 101. Theadvertisement insertion tool 308 can locate the current slice (firstslice) being created by the slicer 106 and can truncate the currentslice to contain video frames (audio samples, or both video frames oraudio samples) up to the start time of the advertisement break 351.

The advertisement insertion tool 308 can insert one or moreadvertisements into the media content in various ways. In oneembodiment, the advertisement insertion tool 308 determines whichadvertisements are to be inserted and inputs them into the slicer 306.If the advertisements are not already sliced, the slicer 106 can slicethe advertisements and output the sliced advertisements with the slicedmedia content 105. The advertisement insertion tool 308 can use the ADS210, for example. In another embodiment, the advertisement insertiontool 308 can track information about the advertisement breaks 351, likewhen the advertisement break 351 is to be inserted, which slices havebeen truncated, or the like. The advertisement insertion tool 308 canoutput the tracked information as metadata 353 to the CMS 118 foradvertisement insertion at the data center 130. In another embodiment,the CMS 118 can generate or modify the descriptor file to indicate theadvertisement breaks in the encoded media content 142 and the individualmedia players 113 can decide which advertisements are to be inserted atthe designated advertisement breaks.

In another embodiment, the advertisement insertion tool 308 receives asecond indication comprising an end time of the advertisement break. Theadvertisement insertion tool 308 identifies a second slice that is thenext slice after the end time of the advertisement break 351. Theadvertisement insertion tool 308 can discard one or more subsequentslices between the first slice and the second slice.

In another embodiment, while the slicer 106 is slicing the media content101 and the advertisement insertion tool 308 receives a first indicationwith a start time for the advertisement break 351, the advertisementinsertion tool 308 can halt the slicing at the start time until a secondindication is received. This second indication designates an end time ofthe advertisement break 351. In a further embodiment, the advertisementinsertion tool 308 receives the second indication that designates theend time of the advertisement break, and instructs the slicer 106 toresume slicing of the media content 101 after the end time of theadvertisement break 351. When resuming slicing of the media content 101,the slicer 106 creates a second slice that is the next slice after theend time of the advertisement break 351.

In another embodiment, the advertisement insertion tool 308 inserts anadvertisement into the stream at the start time of the advertisementbreak 351 and creates a descriptor file comprising the first truncatedslice, the advertisement and the second slice for streaming by the mediaplayer 113. In another embodiment, the advertisement insertion tool 308creates a descriptor file comprising the first truncated slice, thesecond slice and an indication of where an advertisement is to beinserted. The descriptor file may be generated by the advertisementinsertion tool 308, the CMS 118 or elsewhere. The descriptor file can becreated based on a database comprising metadata of the sliced mediacontent as described herein. The media player 113 can read thedescriptor file and determine when to insert advertisements forstreaming the media content. In another embodiment, the truncated slicesthemselves can identify where in the stream advertisements are to beinserted. That is, the descriptor file does not have any additionalindicator to designate an advertisement break other than a truncatedslice in the stream. Alternatively, other techniques may be used todesignate where in the stream the advertisement breaks can be inserted,and the truncated slices are used to accommodate advertisement insertionat those presentation times in the media content.

In another embodiment, the advertisement insertion tool 308 inserts anadvertisement into the stream at the start time of the advertisementbreak. The advertisement insertion tool 308 can update a descriptor fileto insert the advertisement after the first truncated first slice forstreaming by the media player. In one embodiment, the advertisementinsertion tool 308 updates an existing descriptor file to replace thefirst slice with the truncated first slice. That is an entry in thedescriptor file can be updated to reflect that the first slice istruncated. As described above, a new descriptor file may be created withthe truncated file.

In a further embodiment, the descriptor file comprises a second slicethat is the next slice after the first slice. When updating thedescriptor file, the advertisement insertion tool 308 inserts theadvertisement between the first truncated first slice and the secondslice for streaming by the media player.

In another embodiment, the first indication that designates a start timeof the advertisement break 351 also indicates a duration of theadvertisement break 351. The advertisement insertion tool 308 candetermine an end time of the advertisement break 351 from the start timeand the duration. In another embodiment, the first indication designatesa start time and an end time of the advertisement break 351.

Various embodiments described above describe slicing video frames andtruncating the slice to include the frames up to the start time. Inother embodiments, the slice includes video frames and audio samples.When truncating the slice, the advertisement insertion tool 308truncates the first slice to contain the video frames up to the starttime of the advertisement break 351 and the audio samples up to thestart time of the advertisement break 351.

In another embodiment, the first indication may be a start of blackframes and the second indication may be an end of the black frames. Theadvertisement insertion tool 308 may use an automated process todetermine the start of the black frames and the end of the black framesas described herein. In another embodiment, the first indication is astart of an original advertisement in the media content 101 and thesecond indication is an end of the original advertisement. In oneembodiment, the advertisement insertion tool 308 discards one or moresubsequent slices of the original advertisement between the first sliceand the second slice. The first slice may still be truncated toaccommodate the insertion of the advertisement break 351 as describedabove.

In one embodiment, a computing system includes a memory and a processingdevice coupled to the memory. The processing device is configured toexecute the advertisement insertion tool 308. The advertisementinsertion tool 308 performs any of the methodologies described herein.

FIG. 4A is a flow diagram of one embodiment of a method 400 of creatingnew slices to accommodate advertisement insertion. The method 400 isperformed by processing logic that may comprise hardware (circuitry,dedicated logic, etc.), software (such as is run on a general purposecomputing system or a dedicated machine), firmware (embedded software),or any combination thereof. In one embodiment, the advertisementinsertion tool 108 of FIGS. 1-3B performs the method 400. Alternatively,other components of the one or more server computing systems 116 canperform some or all of the operations of method 400.

Referring to FIG. 4A, processing logic begins with determining if apresentation time is received for advertisement insertion (block 402).The presentation time may be received as part of the advertisementinsertion input 109 (or 209) received at the CMS 118. For example, apublisher may playback the media content and may select a point in themedia content to insert the advertisement using a graphical userinterface tool or other means. Sometimes the original media content 101has black-out frames in the video stream where the advertisements shouldbe inserted. Other times the original media content 101 has originaladvertisements in the video stream, but these advertisements need to bereplaced with other advertisements. For example, an over-the-air TV feedmay already have advertisements, but these advertisements were sold forover-the-air TV only. The advertisement insertion tool 108 can receive asignal indicating when the advertisement break starts and another signalindicating when the advertisement break ends in order to insertdifferent advertisements in place of the original advertisements. Thepublisher can easily see the blacked-out frames (or the originaladvertisements) and select the point where the black-out frames beginand end. These selected points can be represented as presentation timesthat are used by the advertisements insertion tool to create new slicesto accommodate the insertion of the advertisement at the appropriatetime. In essence, the blacked-out frames can be removed and replacedwith the advertisement. In one embodiment, the black frames can bedetected by an automated process to designate the playback times foradvertisement insertion. In another embodiment, the processing logicwaits to receive the advertisement insertion input 109 or 209 at block402 to determine when to locate the appropriate slice for insertion andto create the new slices to accommodate the advertisement.

If at block 402 the presentation time is not received, the processinglogic can return until the presentation time is received. If at block402 the presentation time is received, the processing logic locates aslice corresponding to the presentation time (block 404). Once located,the processing logic creates one or more new slices to accommodate theadvertisement (block 406), and updates the descriptor file with the newslices and the advertisement (block 408). Of course, the processinglogic can locate a slice of a single quality profile at block 404, aswell as multiple slices that correspond to multiple quality profiles.That is, the processing logic can locate the corresponding slice in eachof the quality profiles at block 404 and create the new slices for eachof the quality profiles. In this way, the advertisement can be insertedat the same presentation time regardless of the quality profile. Theprocessing logic can also update the descriptor file accordingly.

FIG. 4B is a flow diagram of one embodiment of a method 450 of preparinga stream for inserting one or more slices of an advertisement whenslicing. The method 450 is performed by processing logic that maycomprise hardware (circuitry, dedicated logic, etc.), software (such asis run on a general purpose computing system or a dedicated machine),firmware (embedded software), or any combination thereof. In oneembodiment, the advertisement insertion tool 108 of FIGS. 1-3B performsthe method 450. Alternatively, other components of the one or moreserver computing systems 116 can perform some or all of the operationsof method 450.

Referring to FIG. 4B, processing logic begins with determining if an endof stream is reached (block 452). If not, the processing logicdetermines if a signal is received indicating a start of anadvertisement for advertisement insertion (block 454). If not, theprocessing logic slices the media content as normal (block 456) andreturns to determine if the end of the stream is reached at block 452.At block 454 when the start signal is received, the processing logichalts the slicing and ends the current slice prematurely regardless ofthe time remaining in creating the current slice (block 458). Thistruncated slice indicates where the advertisement is to be inserted. Thetruncated slice can itself identify where the advertisement is to beinserted or a record of the slice can be stored in a database, asdescribed in more detail below. The processing logic waits until itreceives a second signal indicating an end of the advertisement for theadvertisement insertion (block 460). In this embodiment, a second signalis received. In another embodiment, the start and end times of theadvertisement break can be specified and the slicing is halted for theadvertisement break. When the second signal is received at block 460,the processing logic continues the slice with a new slice (block 462).This new slice is the slice to be played back after the advertisement tobe inserted is played back. The processing logic continues back to block452 to determine if the end of the stream is reached. When the end ofthe stream is reached, the processing logic ends the method 450.

As described herein, the presentation time or start time of theadvertisement break can be received as part of the advertisementinsertion input 109 (or 209) received at the CMS 118. For example, apublisher may playback the media content and may select a point in themedia content to insert the advertisement using a graphical userinterface tool or other means. Sometimes the original media content 101has black-out frames in the video stream where the advertisements shouldbe inserted. Other times the original media content 101 has originaladvertisements in the video stream, but these advertisements need to bereplaced with other advertisements.

In another embodiment, the processing logic, when slicing live mediacontent, for example, can receive end slices prematurely, given a signalfrom the content owner or publisher. The signal could be in the livestream or could be received out-of-band via e.g. an API call. Basically,at the time of an advertisement break on a live stream, the processinglogic receives that signal and that would end the current slice, howeverlong it may be. When the processing logic gets a corresponding signal(or other indication) that the advertisement break is finished, theprocessing logic starts a new slice as usual and proceeds from there.The net result would be a list of slices with the following durations 4,4, 4, 2.1, 4, 4, and so on. The list of slices can be used in a firstdescriptor file as described herein. In this example, the one slice witha 2.1 second duration was truncated because the processing logicreceived a signal that an advertisement break was started. In reality,there could have been 30 or 60 or more seconds between this slice andthe subsequent slice because of the advertisement break, but the actualduration does not matter. As described below with respect to FIG. 4C, atthe time of playback, the processing logic can find some existingadvertisements from the user's content library and insert them into thelist of slices. To continue with the example, the processing logic couldend up with a list of files with the following durations 4, 4, 4, 2.1,[4, 4, 4, 4, 4, 1.1,] 4, 4, and so on, where the durations in bracketsare from the files for the advertisement(s) inserted. It should be notedthat the final slice for the advertisement(s) is shorter simply becausethe advertisement's total duration was 21.1 seconds in this example. Thelist of slices can be used in a second descriptor file as describedherein. In these embodiments, the runtime advertisement insertion, atthe time of playback, may be a less CPU-intensive process because theprocessing logic doesn't have to open up the slices and create newslices to insert the advertisements.

For another example, an over-the-air TV feed may already haveadvertisements, but these advertisements were sold for over-the-air TVonly. The advertisement insertion tool 108 can receive a signalindicating when the advertisement break starts and another signalindicating when the advertisement break ends in order to insertdifferent advertisements in place of the original advertisements. Thepublisher can easily see the blacked-out frames (or frames of theoriginal advertisement(s)) and select the point where the black-outframes begin and end. These selected points can be represented aspresentation times that are used by the advertisements insertion tool toend the current slice and start a new slice and to determine where toinsert one or more slices for the advertisements to accommodate theinsertion of the advertisement at the appropriate time. In essence, theblacked-out frames (or frames of the original advertisement(s)) can beremoved and replaced with the advertisement. In one embodiment, theblack frames can be detected by an automated process to designate theplayback times for advertisement insertion. In another embodiment, theprocessing logic waits to receive the advertisement insertion input 109or 209 at block 454 to determine when to prematurely end (truncate) theappropriate slice for advertisement insertion and to start a new sliceafter the advertisement to accommodate the advertisement insertion.

Of course, the processing logic can perform the method on slices of asingle quality stream or on slices of multiple quality profiles. Thatis, the processing logic can slice the media content to accommodate theadvertisement insertion and can insert the slices of the advertisementin the designated locations in each of the different quality profiles.In this way, the advertisement can be inserted at the same presentationtime regardless of the quality profile. The processing logic can alsoupdate the descriptor file accordingly. The advertisement can also besliced into multiple quality profiles to correspond to the differentquality profiles of the stream. Alternatively, the advertisements can beencoded according to a single quality profile or more or less qualityprofiles than the stream itself.

In another embodiment, for VOD at the time of slicing, the user (contentowner or publisher) can specify where advertisement breaks should be.This can come as information in a metadata file or it can trigger offe.g., chapter breaks present in an MP4 file, or other triggerindicators. Either way, as the processing logic slices the media contentand reaches a presentation time for an advertisement break, theprocessing logic can truncate the current slice and then immediatelystart a new slice in the media content. As with live streams, thiseffectively allows the processing logic to skip the pre-processing orthe creation of slices to accommodate advertisements insertions. Putanother way, if a user wanted to insert an advertisement at the exactpoint in time between two slices, the processing logic can list theslices for the content, then list the slices for the advertisement, thenlist the rest of the slices for the content all in the descriptor file144 (e.g., M3U8 file). There would be no need to look for key frames oranything inside the slice(s). If the advertisement is not to be insertedat a slice boundary, by truncating the slice where the ad break couldgo, the processing logic creates an advertisement insertion spot in theslicing of the media content for inserting one or more advertisements,and simply listing the advertisement slices (e.g., URLs for the slicesof the advertisement(s)) in the descriptor file 144 (e.g., M3U8 file).

FIG. 4C is a flow diagram of one embodiment of a method 470 of insertingthe one or more slices of an advertisement into a stream for streaming.The method 470 is performed by processing logic that may comprisehardware (circuitry, dedicated logic, etc.), software (such as is run ona general purpose computing system or a dedicated machine), firmware(embedded software), or any combination thereof. In one embodiment, theadvertisement insertion tool 108 of FIGS. 1-3B performs the method 470.Alternatively, other components of the one or more server computingsystems 116 can perform some or all of the operations of method 470.

Referring to FIG. 4C, processing logic begins with locating a truncatedslice for advertisement insertion (block 472). As described above, thetruncated slice can itself identify where the advertisement is to beinserted since its file may have a duration that is less than thedurations of the other files. Alternatively, a record of the slice wherethe advertisement is to be inserted can be tracked and stored in adatabase. For example, the processing logic can store in the database alist of locations where the advertisement breaks occur in the stream.For example, the database may keep track of the fact that the stream has100 normal duration slices, like 4 seconds, followed by a 1.5 secondslice that indicates where the advertisement(s) is to be inserted, andthen followed by a second number of normal duration slices.

In another embodiment, the processing logic truncates a slice in themedia content (e.g., live stream or VOD asset) at a time of slicing themedia content based on an indication that an advertisement break is tobe inserted at a specified point in the media content. The processinglogic, prior to playback, inserts an advertisement by inserting one ormore advertisement slices into a descriptor file at the specified pointindicated by the truncated slice.

In another embodiment, the processing logic receives advertisementinsertion input identifying a presentation time in media content toinsert an advertisement. In response to the input, the processing logicinspects an original slice of the media content that contains the mediacontent at the presentation time to locate a key frame at or prior tothe presentation time. The processing logic creates a first new slicecomprising the media content of the original slice up to thepresentation time and creates a second new slice comprising the mediacontent of the original slice after the presentation time. Theprocessing logic updates a descriptor file to include the first newslice, the advertisement, and the second new slice for streaming by amedia player executing on a client computing system.

In a further embodiment, the processing logic inspects the originalslice by reading a container comprising encoded video frames andpre-processing the encoded video frames to locate the key frame and alast frame corresponding to the media content at the presentation time.The processing logic creates the first new slice to include the keyframe, any intervening delta frames and the last frame. In oneembodiment, the processing logic copies only the key frame, anyintervening delta frames, and the last frame of the original slicewithout decoding the media content of the container. In anotherembodiment, the processing logic copies the media content of theoriginal slice and discards subsequent frames after the last framecorresponding to the presentation time. The copying and discarding maybe performed without decoding the media content of the container.

In another embodiment, the processing logic creates the second new sliceby decoding the key frame, any intervening delta frames and a firstframe of the media content after the presentation time from the originalslice into raw video. The processing logic re-encodes the raw vide ofthe first frame as a second key frame. In other words, the first framemay be converted from a delta frame to a key frame from the raw videodata. The processing logic creates the second new slice with the secondkey frame and subsequent frames after the first frame until an end ofthe original slice.

In another embodiment, the processing logic creates the new slice withthe second key frame and the subsequent frames by copying the subsequentframes from the original slice without decoding the subsequent frames ofthe original slice. In another embodiment, the processing logic decodesthe subsequent frames of the original slice into raw video andre-encodes the raw video of the subsequent frames of the original slicefor the subsequent frames of the second new slice.

In another embodiment, the processing logic inspects the original sliceby reading a container comprising encoded video and decoding the keyframe, any intervening delta frames and a last frame corresponding tothe media content at the presentation time. The processing logic createsthe first new slice by re-encoding the key frame, any intervening deltaframes and the last frame. In a further embodiment, the processing logiccreates the second new slice by decoding a first frame of the mediacontent after the presentation time from the original slice into rawvideo, re-encoding the raw video of the first frame as a second keyframe. The processing logic creates the new slice with the second keyframe and subsequent frames after the first frame until an end of theoriginal slice.

In another embodiment, the processing logic creates the second new sliceby copying the subsequent frames from the original slice withoutdecoding the subsequent frames of the original slice. In anotherembodiment, the processing logic creates the second new slice bydecoding the subsequent frames of the original slice into raw video andre-encoding the raw video of the subsequent frames of the original slicefor the subsequent frames of the second new slice.

In another embodiment, the processing logic inspects the original sliceby reading a container comprising encoded video frames and encoded audiosamples. The processing logic splits the encoded video frames into afirst processing pipeline and the encoded audio samples into a secondprocessing pipeline. The processing logic, using the first processingpipeline, inspects the original slice of the media content to locate thekey frame.

In another embodiment, the processing logic inspects the original sliceby accessing an index of the media content that identifies key frames ofthe media content. The index may have been previously generated whenpre-processing the media content, such as when added to a contentlibrary by a publisher. Alternatively, the index may be generated inresponse to receiving a first advertisement insertion request oron-the-fly as needed.

In another embodiment, the processing logic receives a first indicationof a start time of where an advertisement is to be inserted into mediacontent. The processing logic also receives a second indicationincluding at least one of an end time of the advertisement or a durationof the advertisement. The first and second indications may be part ofthe same advertisement insertion request, such as described herein. In afurther embodiment, the processing logic receives the duration of theadvertisement, such as in the advertisement insertion request, anddetermines the end time of the advertisement from the start time and theduration. Alternatively, the first indication can be a start of blackframes in the media content and the second indication is an end of theblack frames in the media content. The start of black frames and the endof black frames may be determined by an automated process.Alternatively, a publisher may manually review the media content toidentify the start of the black frames and the end of the black framesas described herein. In another embodiment, the advertisement insertionrequest includes a start time and a duration of an advertisement to beinserted or a start time and an end time for the advertisement. Itshould be noted that the first and second indications can be for ageneric advertisement or for a specific advertisement. For example, theadvertisement insertion can be a placeholder and an ADS, for example,could be used to fill the advertisement insertion based on variousfactors, such as a user profile, geographical locations of therequesting computer, or the like.

In this embodiment, the processing logic inspects a first original sliceof the media content that contains the media content at the start timeto locate a first key frame at or prior to the start time. Theprocessing logic creates a first new slice including the media contentof the first original slice up to the start time. The processing logicalso inspects a second original slice of the media content that containsthe media content at the end time to locate a second key frame at orprior to the end time. The processing logic creates a second new sliceincluding the media content of the second original slice after the endtime until an end of the second original slice.

In a further embodiment, the processing logic updates a descriptor fileto include the first new slice, the advertisement, and the second newslice for streaming by a media player executing on a client computingsystem. The processing logic may update the descriptor file to excludeany intervening slices between the first original slice and the secondoriginal slice when applicable, such as illustrated in FIG. 5B.

In another embodiment, the processing logic inspects the first originalslice and the second original slice by reading a container comprisingencoded video frames and pre-processing the encoded video frames tolocate the first key frame and a last frame corresponding to the mediacontent at the start time and to locate the second key frame and a firstframe corresponding to the media content after the end time. Theprocessing logic creates the first new slice by including the first keyframe, any intervening delta frames and the last frame. The processinglogic creates the second new slice by converting the first frame into athird key frame and creating the second new slice to include the thirdkey frame and subsequent frames after the third key frame to an end ofthe second original slice. In a further embodiment, the processing logiccopies the subsequent frames from the second original slice withoutdecoding the subsequent frames of the second original slice.

FIG. 5A illustrates a stream of slices of media content before and afteradvertisement insertion according to one embodiment. The stream 502 isthe stream before advertisement insertion. In this embodiment, theadvertisement insertion tool 108 receives an advertisement insertioninput 509 that designates where in the stream 502 an advertisementshould be inserted. It should be noted that a slice contains multipleframes, including key frames and delta frames. In this embodiment, eachslice has at least a key frame at the slice boundary, such as atplayback times 0, 4, 8, 12, 16 and so on. The slice may have additionalkey frames, but the advertisement insertion input 509 may notnecessarily correspond to a key frame. As a result, the advertisementinsertion tool 108 creates two new slices to accommodate theadvertisement insertion. To do this, the advertisement insertion tool108 locates the slice that corresponds to a playback time of theadvertisement insertion input 509. In the depicted embodiment, this isslice #2. The advertisement insertion tool 108 may create the first newslice #2A by modifying the slice #2, such as by dropping the data thatcorresponds to the frames after the playback time of the media contentafter the advertisement insertion input 509 (e.g., playback time of 7seconds in the depicted embodiment). In another embodiment, theadvertisement insertion tool 108 may create the first new slice #2A bycopying the corresponding data from the slice #2 that corresponds to theplayback time up to the playback time of the advertisement insertioninput 509 (e.g., playback time of 7 seconds). The advertisementinsertion tool 108 may create the second new slice #2B by locating theprevious key frame before the frame corresponding to the first frame ofthe new slice #2B after the advertisement insertion input 509, anddecoding the key frame and the subsequent frames up to the first frameof the new slice #2B to obtain the raw frame data for the first frame ofthe new slice #2B located at presentation time 36 seconds. Theadvertisement insertion tool 108 can convert this frame into a new keyframe for the new slice #2B. In one embodiment, the advertisementinsertion tool 108 then copies the remaining frames in the slice fromthe original slice #2. In another embodiment, the advertisementinsertion tool 108 could decode and re-encode the remaining frames tothe end of the slice#2, but copying the existing data from slice #2 maybe quicker.

The resulting stream 504 of slices after the advertisement insertion canbe played back with an advertisement (e.g., 30-second advertisement inthe depicted embodiment). Since the advertisement insertion tool 108created the new slice #2B, the media player can request and playback thenew slice #2B after playing the advertisement in a smooth manner. Itshould be noted that the new slice #2B is independently requestable andindependently playable by the media player. It should also be noted thatthe streams 502 and 504 represent a single quality profile of the mediacontent. This process can be repeated for multiple quality profiles.However, once the presentation times and frames have been identified inthe first stream, this information can be used to locate the appropriateframes in the other quality profiles.

In order to stream the stream 504 with the advertisement, theadvertisement insertion tool 108 updates the descriptor file so that themedia player requests the new slices #2A and #2B and the advertisementaccordingly. The following tables illustrate entries of a descriptorfile before and after the advertisement insertion.

PLAYLIST (BEFORE ADVERTISEMENT INSERTION) Slice #1 Slice #2 Slice #3Slice #4 Slice #5 Slice #6

PLAYLIST (AFTER ADVERTISEMENT INSERTION) Slice #1  Slice #2A 30-secAdvertisement  Slice #2B Slice #3 Slice #4 Slice #5 Slice #6

Of course, other slice durations and advertisements can be used as wouldbe appreciated by one of ordinary skill in the art having the benefit ofthis disclosure. Also, more than one advertisement may be inserted intothe stream 502.

The following illustrates an example of a first descriptor file (e.g.,M3U8 file) before advertisement insertion and an example of a seconddescriptor file after the advertisement insertion.

First Descriptor File Before Advertisement Insertion

-   #EXTM3U-   #EXT-X-VERSION:4-   #EXT-X-TARGETDURATION: 5-   #EXT-X-MEDIA-SEQUENCE:0-   #EXTINF:4.0107,-   /content/C00000000.ts-   #EXTINF:4.0107,-   /content/C00000001.ts-   #EXTINF:4.0107,-   /content/C00000002.ts-   #EXTINF:4.0107,-   /content/C00000003.ts-   #EXTINF:4.0107,-   /content/C00000004.ts-   #EXTINF:4.0107,-   /content/C00000005.ts-   #EXTINF:4.0107,-   /content/C00000006.ts-   #EXTINF:4.0107,-   /content/C00000007.ts-   #EXTINF:4.0107,-   /content/C00000008.ts-   #EXTINF:4.0107,-   /content/C00000009.ts-   #EXTINF:4.0107,-   /content/C0000000A.ts-   #EXT-X-ENDLIST    Second Descriptor File after Advertisement Insertion-   #EXTM3U-   #EXT-X-VERSION:4-   #EXT-X-TARGETDURATION: 5-   #EXT-X-MEDIA-SEQUENCE:0-   #EXTINF:4.0107,-   /content/C00000000.ts-   #EXTINF:4.0107,-   /content/C00000001.ts-   #EXTINF:4.0107,-   /content/C00000002.ts-   #EXTINF:4.0107,-   /content/C00000003.ts-   #EXTINF:4.0107,-   /content/C00000004.ts-   #EXT-X-DISCONTINUITY-   #EXTINF:4.0403,-   /ad/C00000000.ts-   #EXTINF:4.0403,-   /ad/C00000001.ts-   #EXTINF:4.0403,-   /ad/C00000002.ts-   #EXTINF:4.0403,-   /ad/C00000003.ts-   #EXTINF:4.0403,-   /ad/C00000004.ts-   #EXTINF:4.0403,-   /ad/C00000005.ts-   #EXTINF:4.0403,-   /ad/C00000006.ts-   #EXTINF:2.2291,-   /ad/C00000007.ts-   #EXT-X-DISCONTINUITY-   #EXTINF:4.0107,-   /content/C00000005.ts-   #EXTINF:4.0107,-   /content/C00000006.ts-   #EXTINF:4.0107,-   /content/C00000007.ts-   #EXTINF:4.0107,-   /content/C00000008.ts-   #EXTINF:4.0107,-   /content/C00000009.ts-   #EXTINF:4.0107,-   /content/C0000000A.tx-   #EXT-X-ENDLIST

The value after the EXTINF tag is the slice's duration. As noted, thelast slice for the advertisement has a duration (e.g., 2 seconds) thatis less than a full-slice duration (e.g., 4 seconds). The firstdescriptor file shows an uninterrupted sequence of slice files. As shownin the second descriptor file, the second descriptor file includes thesame list of slice files, but the list has been split in two with anadvertisement inserted between them. It should be noted that the seconddescriptor file may be the same descriptor file as the first descriptorfile and the contents of the first descriptor file is modified by theadvertisement insertion tool 108 to insert the advertisement.Alternatively, the advertisement insertion tool 108 can create a newdescriptor file with the advertisement inserted to be used in place ofthe first descriptor file as would be appreciated by one of ordinaryskill in the art having the benefit of this disclosure. As anotheralternative embodiment, the descriptor file could be generated on demandas needed by storing the information for the descriptor file in adatabase; instead of modifying a descriptor file to insertadvertisements as described above, the system would generate a differentdescriptor file based on whether or not advertisements should bepresent.

FIG. 5B illustrates a stream of slices of media content before and afteradvertisement insertion according to another embodiment. The stream 552is the stream before advertisement insertion. In this embodiment, theadvertisement insertion tool 108 receives an advertisement insertioninput that designate where in the stream 542 an advertisement should beinserted. In this embodiment, the stream 552 includes black frames thathave been inserted to designate where the advertisements should beinserted. It should be noted that the black frames may represent theactual time available for advertisements to be inserted or may be longeror shorter than the duration of advertisements to be inserted. Apublisher or editor may review the media content and mark the playbacktimes of the start of the black frames and the end of the black frames,and the start time and duration of the black frames or the start and endtimes of the black frames can be input into the advertisement insertiontool 108. In another embodiment, the black frames can be detected by anautomated process to designate the playback times for advertisementinsertion. For convenience, the depicted embodiment shows that the blackframes are 15 seconds in duration and a 15-second advertisement is to beinserted in place of the black frames.

In this embodiment, each slice has at least a key frame at the sliceboundary, such as at playback times 0, 4, 8, 12, 16 and so on. The slicemay have additional key frames, but the black frames may not necessarilycorrespond to the slice boundaries. As a result, the advertisementinsertion tool 108 creates two new slices to accommodate theadvertisement insertion, slices #4A and #10B. However, unlike the stream502 in FIG. 5A in which slice #2 is effectively split into two newstreams #2A and #2B, the stream 552 discards the frames that includeblack frames and creates the new frame #4A for the frames of slice #4before the advertisement insertion and the new frame #10B for the framesof slice #10 after the advertisement insertion. To do this, theadvertisement insertion tool 108 locates the slice that corresponds to aplayback time of the start of the black frames, as well as the playbacktime of the end of the black frames. In the depicted embodiment, theseare slice #4 and slice #10. The advertisement insertion tool 108 maycreate the first new slice #4A by modifying the slice #4, such as bydropping the data that corresponds to the frames after the playback timeof the media content after the start of the black frames (e.g., playbacktime of 13 seconds in the depicted embodiment). In another embodiment,the advertisement insertion tool 108 may create the first new slice #4Aby copying the corresponding data from the slice #4 that corresponds tothe playback time up to the start of the black frames (e.g., playbacktime of 13 seconds). The advertisement insertion tool 108 may create thesecond new slice #10B by locating the previous key frame before theframe corresponding to the first frame of the new slice #10B after theend of the black frames, and decoding the key frame and the subsequentframes up to the first frame of the new slice #10B to obtain the rawframe data for the first frame of the new slice #10B. The advertisementinsertion tool 108 can convert this frame into a new key frame for thenew slice #10B. In one embodiment, the advertisement insertion tool 108then copies the remaining frames in the slice from the original slice#10. In another embodiment, the advertisement insertion tool 108 coulddecode and re-encode the remaining frames to the end of the slice#10,but copying the existing data from slice #10 may be quicker.

The resulting stream 554 of slices after the advertisement insertion canbe played back with an advertisement (e.g., 15-second advertisement inthe depicted embodiment). Since the advertisement insertion tool 108created the new slice #10B, the media player can request and playbackthe new slice #10B after playing the advertisement in a smooth manner.It should be noted that the new slice #10B is independently requestableand independently playable by the media player. It should also be notedthat the streams 552 and 554 represent a single quality profile of themedia content. This process can be repeated for multiple qualityprofiles. However, once the presentation times and frames have beenidentified in the first stream, this information can be used to locatethe appropriate frames in the other quality profiles.

In order to stream the stream 554 with the advertisement, theadvertisement insertion tool 108 updates the descriptor file so that themedia player requests the new slices #4A and #10B and the 15-secondadvertisement accordingly. The following tables illustrate entries of adescriptor file before and after the advertisement insertion.

PLAYLIST (BEFORE ADVERTISEMENT INSERTION) Slice #1 Slice #2 Slice #3Slice #4 Slice #5 Slice #6 Slice #7 Slice #8 Slice #9  Slice #10  Slice#11  Slice #12

PLAYLIST (AFTER ADVERTISEMENT INSERTION) Slice #1 Slice #2 Slice #3 Slice #4A 30-sec Advertisement   Slice #10B  Slice #11  Slice #12

Of course, other slice durations and advertisements can be used as wouldbe appreciated by one of ordinary skill in the art having the benefit ofthis disclosure. Also, more than one advertisement may be inserted intothe stream 552.

FIG. 6 illustrates a diagrammatic representation of a machine in theexemplary form of a computing system 600 for advertisement insertion.Within the computing system 600 is a set of instructions for causing themachine to perform any one or more of the methodologies discussedherein. In alternative embodiments, the machine may be connected (e.g.,networked) to other machines in a LAN, an intranet, an extranet, or theInternet. The machine may operate in the capacity of a server or aclient machine in a client-server network environment, or as a peermachine in a peer-to-peer (or distributed) network environment. Themachine may be a PC, a tablet PC, a set-top-box (STB), a personal dataassistant (PDA), a cellular telephone, a web appliance, a server, anetwork router, switch or bridge, or any machine capable of executing aset of instructions (sequential or otherwise) that specify actions to betaken by that machine. Further, while only a single machine isillustrated, the term “machine” shall also be taken to include anycollection of machines that individually or jointly execute a set (ormultiple sets) of instructions to perform any one or more of themethodologies discussed herein for advertisement insertion, such as themethod 400 described above. In one embodiment, the computing system 600represents various components that may be implemented in the one or moreserver computing systems 116 as described above. Alternatively, theserver computing system 116 may include more or less components asillustrated in the computing system 600.

The exemplary computing system 600 includes a processing device 602, amain memory 604 (e.g., read-only memory (ROM), flash memory, dynamicrandom access memory (DRAM) such as synchronous DRAM (SDRAM), etc.), astatic memory 606 (e.g., flash memory, static random access memory(SRAM), etc.), and a data storage device 616, each of which communicatewith each other via a bus 630.

Processing device 602 represents one or more general-purpose processingdevices such as a microprocessor, central processing unit, or the like.More particularly, the processing device 602 may be a complexinstruction set computing (CISC) microprocessor, reduced instruction setcomputing (RISC) microprocessor, very long instruction word (VLIW)microprocessor, or a processor implementing other instruction sets orprocessors implementing a combination of instruction sets. Theprocessing device 602 may also be one or more special-purpose processingdevices such as an application specific integrated circuit (ASIC), afield programmable gate array (FPGA), a digital signal processor (DSP),network processor, or the like. The processing device 602 is configuredto execute the processing logic (e.g., advertisement insertion 626) forperforming the operations and steps discussed herein.

The computing system 600 may further include a network interface device622. The computing system 600 also may include a video display unit 610(e.g., a liquid crystal display (LCD) or a cathode ray tube (CRT)), analphanumeric input device 612 (e.g., a keyboard), a cursor controldevice 614 (e.g., a mouse), and a signal generation device 620 (e.g., aspeaker).

The data storage device 616 may include a computer-readable storagemedium 624 on which is stored one or more sets of instructions (e.g.,advertisement insertion 626) embodying any one or more of themethodologies or functions described herein. The advertisement insertion626 may also reside, completely or at least partially, within the mainmemory 604 and/or within the processing device 602 during executionthereof by the computing system 600, the main memory 604 and theprocessing device 602 also constituting computer-readable storage media.The advertisement insertion 626 may further be transmitted or receivedover a network via the network interface device 622.

While the computer-readable storage medium 624 is shown in an exemplaryembodiment to be a single medium, the term “computer-readable storagemedium” should be taken to include a single medium or multiple media(e.g., a centralized or distributed database, and/or associated cachesand servers) that store the one or more sets of instructions. The term“computer-readable storage medium” shall also be taken to include anymedium that is capable of storing a set of instructions for execution bythe machine and that causes the machine to perform any one or more ofthe methodologies of the present embodiments. The term“computer-readable storage medium” shall accordingly be taken toinclude, but not be limited to, solid-state memories, optical media,magnetic media or other types of mediums for storing the instructions.The term “computer-readable transmission medium” shall be taken toinclude any medium that is capable of transmitting a set of instructionsfor execution by the machine to cause the machine to perform any one ormore of the methodologies of the present embodiments.

The advertisement module 632, components, and other features describedherein (for example in relation to FIGS. 4A, 4B and 4C) can beimplemented as discrete hardware components or integrated in thefunctionality of hardware components such as ASICS, FPGAs, DSPs, orsimilar devices. The advertisement module 632 may implement operationsof advertisement insertion as described herein with respect to FIG. 4A,4B or FIG. 4C. In addition, the advertisement module 632 can beimplemented as firmware or functional circuitry within hardware devices.Further, the advertisement module 632 can be implemented in anycombination hardware devices and software components.

The foregoing description, for purpose of explanation, has beendescribed with reference to specific embodiments. However, theillustrative discussions above are not intended to be exhaustive or tolimit the invention to the precise forms disclosed. Many modificationsand variations are possible in view of the above teachings. Theembodiments were chosen and described in order to best explain theprinciples of the invention and its practical applications, to therebyenable others skilled in the art to utilize the invention and variousembodiments with various modifications as may be suited to theparticular use contemplated.

What is claimed is:
 1. A method comprising: identifying media contentcomprising a stream of slices; receiving, by an advertisement insertiontool executing by a processing device, a first indication of a starttime of an advertisement break to be inserted into the stream of slicesof the media content; identifying, from the stream of slices a firstslice that encompasses the start time of the advertisement break;inspecting a container of the first slice to locate key frames, whereinthe container comprises metadata information and encoded video frames,wherein the inspecting the container is performed without decoding theencoded video frames; creating a modified slice that contains theencoded video frames up to the start time of the advertisement break anddoes not contain the encoded video frames after the start time of theadvertisement break; and creating a new slice that contains encodedvideo frames after an end time of the advertisement break, whereincreating the new slice comprises: locating a preceding key frame priorin time to the end time; decoding the preceding key frame and anyintervening video frames between the preceding key frame and a frame atthe end time; and creating, from the decoded frame at the end time, anew key frame for the new slice.
 2. The method of claim 1, furthercomprising: receiving, by the advertisement insertion tool, a secondindication comprising the end time of the advertisement break;identifying a second slice of the stream of slices that contains the endtime of the advertisement break, wherein the second slice is the newslice; and discarding any one or more subsequent slices between thefirst slice and the second slice.
 3. The method of claim 1, furthercomprising: slicing the media content into the stream of slices, whereinthe receiving the first indication comprises receiving the firstindication while slicing the media content; and halting the slice of themedia content at the start time until a second indication is received,wherein the second indication comprises the end time of theadvertisement break.
 4. The method of claim 3, further comprising:receiving, by the advertisement insertion tool, the second indicationcomprising the end time of the advertisement break; and resuming theslicing of the media content after the end time of the advertisementbreak, wherein the resuming comprises creating the new slice.
 5. Themethod of claim 4, further comprising: inserting an advertisement intothe stream at the start time of the advertisement break; and creating adescriptor file comprising the modified slice, the advertisement and thenew slice for streaming by a media player executing on a clientcomputing system.
 6. The method of claim 1, further comprising insertingan advertisement into the stream at the start time of the advertisementbreak.
 7. The method of claim 6, further comprising: updating adescriptor file to insert the advertisement after the modified slice forstreaming by a media player executing on a client computing system; andupdating the descriptor file to replace the first slice with themodified slice.
 8. The method of claim 1, wherein the receiving thefirst indication further comprising receiving a duration of theadvertisement break, and wherein the method further comprisesdetermining the end time of the advertisement break from the start timeand the duration.
 9. The method of claim 1, wherein the first slicecomprises audio samples corresponding to the video frames, and whereincreating the modified slice comprises creating the first slice tocontain the encoded video frames up to the start time of theadvertisement break and the audio samples up to the start time of theadvertisement break.
 10. The method of claim 1, wherein the firstindication is a start of black frames in the media content.
 11. Themethod of claim 10, wherein the start of the black frames is determinedby an automated process.
 12. The method of claim 1, wherein the firstindication is a start of an original advertisement in the media content.13. The method of claim 1, wherein creating the new slice comprisescreating the new slice to contain the encoded video frames of the firstslice after the start time of the advertisement break.
 14. The method ofclaim 1, wherein the start time of the advertisement break is not on aslice boundary.
 15. A non-transitory computer readable storage mediumincluding instructions that, when executed by a processing device, causethe processing device to perform operations comprising: identifyingmedia content comprising a stream of slices; receiving, by anadvertisement insertion tool executing by a processing device, a firstindication of a start time of an advertisement break to be inserted intothe stream of slices of the media content; identifying, from the streamof slices a first slice that encompasses the start time of theadvertisement break: inspecting a container of the first slice to locatekey frames, wherein the container comprises metadata information andencoded video frames, wherein the inspecting the container is performedwithout decoding the encoded video frames; creating a modified slicethat contains the encoded video frames up to the start time of theadvertisement break and does not contain the encoded video frames afterthe start time of the advertisement break; and creating a new slice thatcontains encoded video frames after an end time of the advertisementbreak, wherein creating the new slice comprises: locating a precedingkey frame prior in time to the end time; decoding the preceding keyframe and any intervening video frames between the preceding key frameand a frame at the end time; and creating, from the decoded frame at theend time, a new key frame for the new slice.
 16. The non-transitorycomputer readable storage medium of claim 15, wherein the operationsfurther comprise: receiving, by the advertisement insertion tool, asecond indication comprising the end time of the advertisement break;identifying a second slice of the stream of slices that contains the endtime of the advertisement break, wherein the second slice is a differentslice than the first slice; and discarding any one or more subsequentslices between the first slice and the second slice.
 17. Thenon-transitory computer readable storage medium of claim 15, wherein theoperations further comprise: slicing the media content into the streamof slices, wherein the receiving the first indication comprisesreceiving the first indication while slicing the media content; andhalting the slice of the media content at the start time until a secondindication is received, wherein the second indication comprises the endtime of the advertisement break.
 18. A computing system comprising: amemory; and a processing device, coupled to the memory, wherein theprocessing device is configured to execute an advertisement insertiontool, wherein the advertisement insertion tool is to: identify mediacontent comprising a stream of slices; receive a first indication of astart time of an advertisement break to be inserted into the stream ofslices of the media content; identify, from the stream of slices a firstslice that encompasses the start time of the advertisement break;identifying, from the stream of slices a first slice that encompassesthe start time of the advertisement break; create a modified slice thatcontains the encoded video frames up to the start time of theadvertisement break and does not contain the encoded video frames afterthe start time of the advertisement break: and create a new slice thatcontains encoded video frames after an end time of the advertisementbreak, wherein the advertisement insertion creates the new slice bylocating a preceding key frame prior in time to the end time, decodingthe preceding key frame and any intervening video frames between thepreceding key frame and a frame at the end time, and creating, from thedecoded frame at the end time, a new key frame for the new slice. 19.The computing system of claim 18, wherein advertisement tool is furtherconfigured to: receive a second indication comprising end time of theadvertisement break; identify a second slice of the stream of slicesthat contains the end time of the advertisement break, wherein thesecond slice is a different slice than the first slice; and discard anyone or more subsequent slices between the first slice and the secondslice.
 20. The computing system of claim 18, wherein advertisement toolis further configured to: slice the media content into the slices untilreceiving the first indication; and halt the slice of the media contentat the start time until a second indication is received, wherein thesecond indication comprises the end time of the advertisement break.